Skip to content

[#589] feat: Adding yugabyteDB as a supported connector #590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[#589] feat: Adding yugabyteDB as a supported connector #590

wants to merge 4 commits into from

Conversation

@gargsans-yb
Copy link

@gargsans-yb gargsans-yb commented Aug 31, 2025 •
edited
Loading

Adding yugabyteDB as part of supported source and destination connector for unstructured-ingest

To add yugabyteDB connector:

uv add "unstructured-ingest[yugabytedb]"

This installs yugabytedb specific driver psycopg2-yugabytedb which has features like setting load_balancer and topology specific to yugabyteDB.

GH Issue: #589

@gargsans-yb
[GROWENG-124][GROWENG-126] Adding yugabytedb as connector in
1896c7e 
unstructured.io

Summary: Adding yugabytedb as a source and destination type connector
for unstructured.io

Reviewers: amogh-yb
@gargsans-yb
Merge pull request #1 from yugabyte/sgarg/add-ybdb-connector-to-unstr…
22400c7 
…uctured

[GROWENG-124][GROWENG-126] Adding yugabytedb as connector in unstructured.io
@gargsans-yb
Copy link
Author

gargsans-yb commented Sep 3, 2025

Hi @bryan-unstructured ,
Could you please take a look at our request to add YugabyteDB as a connector?
We noticed your recent commits and thought you'd be the best person to ask. Please let us know if we need to contact someone else for this.
Thanks

@bryan-unstructured
Copy link
Contributor

bryan-unstructured commented Sep 4, 2025 •
edited
Loading

Hi @bryan-unstructured , Could you please take a look at our request to add YugabyteDB as a connector? We noticed your recent commits and thought you'd be the best person to ask. Please let us know if we need to contact someone else for this. Thanks

Hi I'm no longer working on the project.
@ron-unstructured can you please take a look at user's request?

@gargsans-yb
Copy link
Author

gargsans-yb commented Sep 9, 2025

Hi @ron-unstructured, were you able to go through the request?
Thanks.

@gargsans-yb
Copy link
Author

gargsans-yb commented Sep 15, 2025

Hey @ron-unstructured, just checking up on this request to add YugabyteDB as a connector.
Thanks.

@gargsans-yb
Copy link
Author

gargsans-yb commented Oct 14, 2025

Hi @potter-potter , since you were the reviewer of some of the last committed PRs, could you please take a look at our request?
Thanks.

@gargsans-yb
Merge branch 'Unstructured-IO:main' into main
4187a80
@gargsans-yb
[GROWENG-124][GROWENG-126] Adding yugabytedb as connector in
3690df0 
unstructured.io

Summary:
* Adding tests for yugabytedb connector
* Added set-up for yugabytedb as a source and destination connector
* Added expected results for yugabytedb as source, destination and
stager
@gargsans-yb gargsans-yb force-pushed the sgarg/add-ybdb-connector-to-unstructured branch from bac1b2e to 3690df0 Compare November 10, 2025 09:12
@gargsans-yb
Copy link
Author

gargsans-yb commented Nov 10, 2025

Hi @potter-potter , a reminder for this PR.
We have now added tests for yugabytedb connector, similar to the ones present for other connectors in unstructured. PTAL.
The tests ran have passed:

python -m pytest test/integration/connectors/sql/test_yugabytedb.py -v

=========================================================================================================== test session starts ===========================================================================================================
platform linux -- Python 3.11.11, pytest-8.4.2, pluggy-1.6.0 -- /home/gargsans/gen-ai/unstructured-ingest/.venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.11.11', 'Platform': 'Linux-4.18.0-553.34.1.el8_10.x86_64-x86_64-with-glibc2.28', 'Packages': {'pytest': '8.4.2', 'pluggy': '1.6.0'}, 'Plugins': {'asyncio': '1.2.0', 'timeout': '2.4.0', 'Faker': '37.12.0', 'pytest_tagging': '1.6.0', 'mock': '3.15.1', 'metadata': '3.1.1', 'check': '2.6.0', 'json-report': '1.5.0', 'cov': '7.0.0'}}
tagging: tags=[] , exclude-tags=[]
rootdir: /home/gargsans/gen-ai/unstructured-ingest
configfile: pyproject.toml
plugins: asyncio-1.2.0, timeout-2.4.0, Faker-37.12.0, pytest_tagging-1.6.0, mock-3.15.1, metadata-3.1.1, check-2.6.0, json-report-1.5.0, cov-7.0.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 4 items                                                                                                                                                                                                                         

test/integration/connectors/sql/test_yugabytedb.py::test_yugabytedb_source PASSED                                                                                                                                                   [ 25%]
test/integration/connectors/sql/test_yugabytedb.py::test_yugabytedb_destination PASSED                                                                                                                                              [ 50%]
test/integration/connectors/sql/test_yugabytedb.py::test_yugabytedb_stager[upload_file_ndjson] PASSED                                                                                                                               [ 75%]
test/integration/connectors/sql/test_yugabytedb.py::test_yugabytedb_stager[upload_file] PASSED                                                                                                                                      [100%]

============================================================================================================ warnings summary =============================================================================================================
test/integration/connectors/sql/test_yugabytedb.py::test_yugabytedb_source
test/integration/connectors/sql/test_yugabytedb.py::test_yugabytedb_destination
  /home/gargsans/gen-ai/unstructured-ingest/unstructured_ingest/interfaces/connector.py:37: PydanticDeprecatedSince211: Accessing the 'model_fields' attribute on the instance is deprecated. Instead, you should access this attribute from the model class. Deprecated in Pydantic V2.11 to be removed in V3.0.
    access_config_type = self.model_fields["access_config"].annotation

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================================ 4 passed, 2 warnings in 227.83s (0:03:47) ================================================================================================

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@gargsans-yb @bryan-unstructured